A linear programming methodology for approximate dynamic programming

[EN] The linear programming (LP) approach to solve the Bellman equation in dynamic programming is a well-known option for finite state and input spaces to obtain an exact solution. However, with function approximation or continuous state spaces, refinements are necessary. This paper presents a methodology to make approximate dynamic programming via LP work in practical control applications with continuous state and input spaces. There are some guidelines on data and regressor choices needed to obtain meaningful and well-conditioned value function estimates. The work discusses the introduction of terminal ingredients and computation of lower and upper bounds of the value function. An experimental inverted-pendulum application will be used to illustrate the proposal and carry out a suitable comparative analysis with alternative options in the literature.The authors are grateful for the financial support of the Spanish Ministry of Economy and the European Union, grant DPI2016-81002-R (AEI/FEDER, UE), and the PhD grant from the Government of Ecuador (SENESCYT).Diaz, H.; Sala, A.; Armesto Ángel, L. (2020). A linear programming methodology for approximate dynamic programming. International Journal of Applied Mathematics and Computer Science (Online). 30(2):363-375. https://doi.org/10.34768/amcs-2020-0028S36337530

Decentralized Multi-Agent Formation Control with Pole-Region Placement via Cone-Complementarity Linearization

[EN] An output-feedback decentralised formation control strategy is pursued under pole-region constraints, assuming that the agents have access to relative position measurements with respect to a set of neighbors in a graph describing the sensing topology. No communication between the agents is assumed; however, a shared one-way communication channel with a pilot is needed for steering tasks. Each agent has a separate copy of the same controller. A virtual structure approach is presented for the formation steering as a whole; actual formation control is established via cone-complementarity linearization algorithms for the appropriate matrix inequalities. In contrast to other research where only stable consensus is pursued, the proposed method allows us to specify settling-time, damping and bandwidth limitations via pole regions. In addition, a full methodology for the decoupled handling of steering and formation control is provided. Simulation results in the example section illustrate the approachThe first author is grateful for the financial support via the grant GVA/2021/082 from Generalitat Valenciana. Part of the authors' research activity in related topics is funded via the grant PID2020-116585GB-I00 through MCIN/AEI/10.13039/501100011033 and by the European Union.González Sorribes, A.; Sala, A.; Armesto, L. (2022). Decentralized Multi-Agent Formation Control with Pole-Region Placement via Cone-Complementarity Linearization. International Journal of Applied Mathematics and Computer Science. 32(3):415-428. https://doi.org/10.34768/amcs-2022-003041542832

On improving robot image-based visual servoing based on dual-rate reference filtering control strategy

It is well known that the use of multi-rate control techniques have improved the performance of many systems in general, and robotic systems, in particular. The main contribution of this paper is the generalization of the Reference Filtering control strategy from a dual-rate point of view, improving its inherent properties by overcoming the problem of sensor latency. In the paper, we discuss and analyze the improvements introduced by the novel dual-rate reference filtering control strategy in terms of convergence time, reachability and robustness. More specifically, we discuss the capability to solve positioning tasks, when hardware limitations are present with large sampling rates. In addition, a comparison is made between the single-rate and the proposed dual-rate control strategies to prove the advantages of the latter approach. A complete set-up has been prepared for validation, including a six degree of freedom (DOF) industrial manipulator, a smart camera and embedded hardware used as a high level controller.This work was supported by VALi+d Program (Generalitat Valenciana), DIVISAMOS Project (Spanish Ministry, DPI-2009-14744-C03-01), PROMETEO Program (Conselleria d'Educacio, Generalitat Valenciana) and SAFEBUS: Ministry of Economy and Competitivity, IPT-2011-1165-370000).Solanes Galbis, JE.; Muñoz Benavent, P.; Girbés, V.; Armesto Ángel, L.; Tornero Montserrat, J. (2015). On improving robot image-based visual servoing based on dual-rate reference filtering control strategy. Robotica. 1-18. https://doi.org/10.1017/S0263574715000454S11

Volume-weighted Bellman error method for adaptive meshing in approximate dynamic programming

[EN] Optimal control and reinforcement learning have an associate “value function” which must be suitably approximated. Value function approximation problems usually have different precision requirements in different regions of the state space. An uniform gridding wastes resources in regions in which the value function is smooth, and, on the other hand, has not enough resolution in zones with abrupt changes. The present work proposes an adaptive meshing methodology in order to adapt to these changing requirements without incrementing too much the number of parameters of the approximator. The proposal is based on simplicial meshes and Bellman error, with a criteria to add and remove points from the mesh: modifications to proposals in earlier literature including the volume of the affected simplices are proposed, alongside with methods to manipulate the mesh triangulation.[ES] El control óptimo y aprendizaje por refuerzo lleva asociada una "función de valor'' que debe ser adecuadamente aproximada. Estos problemas de aproximar funciones de valor tienen, usualmente, diferentes requerimientos de precisión en diferentes regiones del espacio de estados. Un mallado uniforme tiene problemas porque desperdicia recursos en regiones en las que la función de valor es suave, mientras que no tiene la suficiente resolución en zonas con grandes cambios en dicha función.  El presente trabajo propone una metodología de programación dinámica aproximada con mallado adaptativo, para poder adaptarse a dichos requerimientos cambiantes sin incrementar en exceso el número de parámetros del aproximador. La propuesta se basa en mallados simpliciales y en el error en la ecuación de Bellman con un criterios para añadir y quitar puntos del mallado: se modificarán propuestas de la literatura incluyendo el volumen de los símplices afectados en los criterios, y se detallarán las manipulaciones de la triangulación necesarias.Este artículo ha sido financiado por la Agencia Española de Investigación mediante el proyecto del Plan Nacional PID2020-116585GB-I00.Armesto, L.; Sala, A. (2021). Método de error de Bellman con ponderación de volumen para mallado adaptativo en programación dinámica aproximada. Revista Iberoamericana de Automática e Informática industrial. 19(1):37-47. https://doi.org/10.4995/riai.2021.15698OJS3747191Albertos, P., Sala, A., 2006. Multivariable control systems: an engineering approach. Springer, London, U.K.Allgower, F., Zheng, A., 2012. Nonlinear model predictive control.Antos, A., Szepesvári, C., Munos, R., 2008. Learning near optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning 71 (1), 89-129. https://doi.org/10.1007/s10994-007-5038-2Ariño, C., Pérez, E., Querol, A., Sala, A., 2014. Model predictive control for discrete fuzzy systems via iterative quadratic programming. In: Fuzzy Systems (FUZZ-IEEE), 2014 IEEE International Conference on. IEEE, pp. 2288-2293. https://doi.org/10.1109/FUZZ-IEEE.2014.6891633Ariño, C., Pérez, E., Sala, A., 2010. Guaranteed cost control analysis and iterative design for constrained takagi-sugeno systems. Engineering Applications of Artificial Intelligence 23 (8), 1420-1427. https://doi.org/10.1016/j.engappai.2010.03.004Armesto, L., Girbés, V., Sala, A., Zima, M., Smídl, V., 2015. Duality-based nonlinear quadratic control: Application to mobile robot trajectory-following. IEEE Transactions on Control Systems Technology 23 (4), 1494-1504. https://doi.org/10.1109/TCST.2014.2377631Athans, M., Falb, P. L., 2013. Optimal control: an introduction to the theory and its applications. Courier Corporation.Bertsekas, D. P., 2018. Abstract dynamic programming. Athena Scientific.Bertsekas, D. P., Tsitsiklis, J. N., 1996. Neuro-Dynamic Programming. Athena Scientific, Belmont, MA, USA.Busoniu, L., Babuska, R., De Schutter, B., Ernst, D., 2010. Reinforcement learning and dynamic programming using function approximators. CRC press, Boca Raton, FL, USA.Busoniu, L., Ernst, D., De Schutter, B., Babuska, R., 2010. Approximate dynamic programming with a fuzzy parameterization. Automatica 46 (5), 804-814. https://doi.org/10.1016/j.automatica.2010.02.006Camacho, E. F., Bordons, C., 2010. Control predictivo: Pasado, presente y futuro. Revista Iberoamericana de Automática e Informática Industrial 1 (3), 5-28.De Farias, D. P., Van Roy, B., 2003. The linear programming approach to approximate dynamic programming. Operations research 51 (6), 850-865. https://doi.org/10.1287/opre.51.6.850.24925Deisenroth, M. P., Neumann, G., Peters, J., et al., 2013. A survey on policy search for robotics. Foundations and Trends in Robotics 2 (1-2), 1-142. https://doi.org/10.1561/2300000021Díaz, H., Armesto, L., Sala, A., 2019. Metodología de programación dinámica aproximada para control óptimo basada en datos. Revista Iberoamericana de Automática e Informática industrial 16 (3), 273-283. https://doi.org/10.4995/riai.2019.10379Díaz, H., Armesto, L., Sala, A., 3 2020. Fitted Q-function control methodology based on takagi-sugeno systems. IEEE Transactions on Control Systems Technology 28 (2), 477-488. https://doi.org/10.1109/TCST.2018.2885689Díaz, H., Sala, A., Armesto, L., 2020. A linear programming methodology for approximate dynamic programming. International Journal of Applied Mathematics and Computer Science 30 (2).Duarte-Mermoud, M., Milla, F., 2018. Estabilizador de sistemas de potencia usando control predictivo basado en modelo. Revista Iberoamericana de Automática e Informática industrial. https://doi.org/10.4995/riai.2018.10056Fairbank, M., Alonso, E., 6 2012. The divergence of reinforcement learning algorithms with value-iteration and function approximation. In: The 2012 International Joint Conference on Neural Networks (IJCNN). pp. 1-8. https://doi.org/10.1109/IJCNN.2012.6252792Grüne, L., 1997. An adaptive grid scheme for the discrete hamilton-jacobibellman equation. Numerische Mathematik 75, 319-337. https://doi.org/10.1007/s002110050241Hornik, K., Stinchcombe, M., White, H., 1989. Multilayer feedforward networks are universal approximators. Neural Networks 2 (5), 359 - 366. https://doi.org/10.1016/0893-6080(89)90020-8Inc, T. M., 2021. Matlab delaunay documentation. URL: https://www.mathworks.com/help/matlab/ref/delaunay.htmlLewis, F. L., Liu, D., 2013. Reinforcement learning and approximate dynamic programming for feedback control. Wiley, Hoboken, NJ, USA.https://doi.org/10.1002/9781118453988Lewis, F. L., Vrabie, D., 2009. Reinforcement learning and adaptive dynamic programming for feedback control. Circuits and Systems Magazine, IEEE 9 (3), 32-50. https://doi.org/10.1109/MCAS.2009.933854Li, W., Todorov, E., 2007. Iterative linearization methods for approximately optimal control and estimation of non-linear stochastic system. International Journal of Control 80 (9), 1439-1453. https://doi.org/10.1080/00207170701364913Liberzon, D., 2011. Calculus of variations and optimal control theory: a concise introduction. Princeton university press. https://doi.org/10.2307/j.ctvcm4g0sMunos, R., Moore, A., 2002. Variable resolution discretization in optimal control. Machine learning 49 (2-3), 291-323. https://doi.org/10.1023/A:1017992615625Rubio, F. R., Navas, S. J., Ollero, P., Lemos, J. M., Ortega, M. G., 2018. Control óptimo aplicado a campos de colectores solares distribuidos. Revista Iberoamericana de Automática e Informática industrial.Santos, M., 2011. Un enfoque aplicado del control inteligente. Revista Iberoamericana de Automática e Informática Industrial RIAI 8 (4), 283-296. https://doi.org/10.1016/j.riai.2011.09.016Sherstov, A. A., Stone, P., 2005. Function approximation via tile coding: Automating parameter choice. In: International Symposium on Abstraction, Reformulation, and Approximation. Springer, pp. 194-205. https://doi.org/10.1007/11527862_14Sutton, R. S., Barto, A. G., 1998. Reinforcement learning: An introduction. Vol. 1. MIT press Cambridge.Ziogou, C., Papadopoulou, S., Georgiadis, M. C., Voutetakis, S., 2013. On-line nonlinear model predictive control of a pem fuel cell system. Journal of Process Control 23 (4), 483-492. https://doi.org/10.1016/j.jprocont.2013.01.01

Fitted Q-Function Control Methodology Based on Takagi-Sugeno Systems

"© 2020 IEEE. Personal use of this material is permitted. Permissíon from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertisíng or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works."[EN] This paper presents a combined identification/ Q-function fitting methodology that involves identification of a Takagi-Sugeno model, computation of (sub)optimal controllers from linear matrix inequalities (LMIs), and subsequent data-based fitting of the Q-function via monotonic optimization. The LMI-based initialization provides a conservative solution, but it is a sensible starting point to avoid convergence/local-minima issues in raw data-based fitted Q-iteration or Bellman residual minimization. An inverted-pendulum experimental case study illustrates the approach.This work was supported in part by the Spanish Ministry of Economy and European Union (AEI/FEDER, UE) under Grant DPI2016-81002-R and in part by the Government of Ecuador through the Ph.D. Grant SENESCYT.Diaz-Iza, HP.; Armesto, L.; Sala, A. (2020). Fitted Q-Function Control Methodology Based on Takagi-Sugeno Systems. IEEE Transactions on Control Systems Technology. 28(2):477-488. https://doi.org/10.1109/TCST.2018.2885689S47748828

Probabilistic Self-Localization and Mapping: An Asynchronous Multirate Approach

"© 2008 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works."[EN] In this paper, we present a set of robust and efficient algorithms with O(N) cost for the solution of the Simultaneous Localization And Mapping (SLAM) problem of a mobile robot. First, we introduce a novel object detection method, which is mainly based on multiple line fitting method for landmark detection with regular constrained angles. Second, a line-based pose estimation method is proposed, based on LeastSquares (LS). This method performs the matching of lines, providing the global pose estimation under assumption of known Data-Association. Finally, we extend the FastSLAM (FActored Solution To SLAM) algorithm for mobile robot self-localisation and mapping by considering the asynchronous sampling of sensors and actuators. In this sense, multi-rate asynchronous holds are used to interface signals with different sampling rates. Moreover, an asynchronous fusion method to predict and update mobile robot pose and map is also presented. In addition to this, FastSLAM 1.0 has been also improved by considering the estimated pose with the LS-approach to re-allocate each particle of the posterior distribution of the robot pose. This approach has a lower computational cost than the original Extended Kalman Filtering (EKF) approach in FastSLAM 2.0. All these methods have been combined in order to perform an efficient and robust self-localization and map building process. Additionally, these methods have been validated with experimental real data, in mobile robot moving on an unknown environment for solving the SLAM problem.This work has been supported by the Spanish Government (MCyT) research project BIA2005-09377-C03-02 and by the Italian Government (MIUR) research project PRIN2005097207.Armesto, L.; Ippoliti, G.; Longhi, S.; Tornero Montserrat, J. (2008). Probabilistic Self-Localization and Mapping: An Asynchronous Multirate Approach. IEEE Robotics & Automation Magazine. 15(2):77-88. https://doi.org/10.1109/M-RA.2007.907355S778815

Low-cost Printable Robots in Education

The final publication is available at Springer via http://dx.doi.org/10.1007/s10846-015-0199-xThe wider availability of 3D printing has enabled small printable robots (or printbots) to be incorporated directly into engineering courses. Printbots can be used in many ways to enhance lifelong learning skills, strengthen understanding and foster teamwork and collaboration. The experiences outlined in this paper were used in our teaching during the last academic year, although much of the methodology and many of the activities have been used and developed over the past 8 years. They include project based assignments carried out by multidisciplinary and multicultural teams, a number of theoretical and practical classroom and laboratory activities all aimed at familiarizing students with fundamental concepts, programming and simulation, and which now form part of our regular robotics courses, and some brief descriptions of how printable robots are being used by students carrying out final projects for Bachelor and Master degrees. The online resources show many of these activities in action.Armesto Ángel, L.; Fuentes-Durá, P.; Perry, DR. (2016). Low-cost Printable Robots in Education. Journal of Intelligent and Robotic Systems. 81(1):5-24. doi:10.1007/s10846-015-0199-xS524811Criteria for accrediting engineering programs (Unknown Month 2015, 2014). http://www.abet.org/eac-criteria-2014-2015Board, N.S.: Moving forward to improve engineering education (2007). http://www.nsf.gov/pubs/2007/nsb07122/nsb07122.pdfCampion, G., Bastin, G., d’Andréa Novel, B.: Structural properties and classification of kinematic and dynamic models of wheeled mobile robots. IEEE Trans. Robot. Autom. 12(1), 47–62 (1996)Carberry, A.R., Lee, H.-S., Ohland, M.W.: Measuring engineering design self-efficacy. J. Eng. Educ. 99(1), 71–79 (2010)Castro. A.: Robotic arm with 6 dof (2012). http://www.thingiverse.com/thing:30163Choset, H., Lynch, K.M., Hutchinson, S., Kantor, G.A., Burgard, W., Kavraki, L.E., Thrun, S.: Principles of Robot Motion: Theory, Algorithms, and Implementations. MIT Press, Cambridge MA (2005)d’Andréa Novel, B., Campion, G., Bastin, G.: Control of nonholonomic wheeled mobile robots by state feedback linearization. Int. J. Robot. Res. 14(6), 543–559 (1995)Denavit, J., Hartenberg, R.S.: A kinematic notation for lower-pair mechanisms based on matrices. Trans. ASME J. Appl. Mech 22(2), 215–221 (1955)Dowdall. J.: Rofi robot five (2012). http://www.projectbiped.com/prototypes/rofiEliot, M., Howard, P., Nouwens, F., Stojcevski, A., Mann, L., Prpic, J., Gabb, R., Venkatesan, S., Kolmos, A.: Developing a conceptual model for the effective assessment of individual student learning in team-based subjects. Australas. J. Eng. Educ. 18(1), 105–112 (2012)Fox, D., Burgard, W., Thrun, S.: The dynamic window approach to collision avoidance. Robot. Autom. Mag. IEEE 4(1), 23–33 (1997)Fuentes-Dura, P., Armesto, L., Perry, D.: Multidisciplinary projects: Critical points and perceptions in valladolid in innovation and quality in engineering education. In: Innovation and Quality in Engineering Education, pp 315–331 (2012)Fuentes-Dura, P., Cazorla, M.P., Molina, M.G., Perry, D.: European project semester: Good practices for competence acquisition. In: Valencia Global, pp 165– 172 (2014)González, J., Barrientos, A., Prieto-Moreno, A., de Frutos, M.A.: Miniskybot 2 (2012). http://www.iearobotics.com/wiki/index.php?Miniskybot_2Gonzalez-Gomez, J., Valero-Gomez, A., Prieto-Moreno, A., Abderrahim, M.: A new open source 3d-printable mobile robotic platform for education. In: Rckert, U., Joaquin, S., Felix, W. (eds.) Advances in Autonomous Mini Robots, pp 49–62. Springer, Berlin Heidelberg (2012)Gonzlez, J., Wagenaar, R. (eds.): Tuning Educational Structures in Europe University of Deusto and Groningen. Deusto (2003)Heinrich, E., Bhattacharya, M., Rayudu, R.: Preparation for lifelong learning using eportfolios. Eur. J. Eng. Educ. 32(6), 653–663 (2007)Khatib, O.: Real-time obstacle avoidance for manipulators and mobile robots. The Int. J. Robot. Res. 5(1), 90–98 (1986)Krassman, J.: Quadcopter hummingbird ii (2013). http://www.thingiverse.com/thing:167721Langevin, G.: Inmoov (2012). http://www.inmoov.frMadox: ecanum wheel rover 2 (2011). http://www.madox.net/blog/2011/01/24/mecanum-wheel-rover-2Miles, M.B., Analysis, A.M.: Huberman. Qualitative Data: An Expanded Sourcebook. SAGE Publications (1994)Minguez, J., Montano, L.: Nearness diagram (nd) navigation: Collision avoidance in troublesome scenarios. IEEE Trans. Robot. Autom. 20, 2004 (2004)Olalla: Caterpillator v1.1 (2011). http://www.thingiverse.com/thing:8559Ollero, A.: Robótica. Manipuladores y robots móviles Marcombo, S.A. Barcelona (2001)Price, M.: Hf08 hexapod robot (2012). http://www.heliumfrog.com/hf08robot/hf08blog.htmlRawat, K., Massiha, G.: A hands-on laboratory based approach to undergraduate robotics education. In: Proceedings of 2004 IEEE International Conference on Robotics and Automation 2, pp 1370–1374 (2004)Robotics, C.: Virtual experimentation robotic platform (v-rep) (2013). www.coppeliarobotics.comScott, B.: Principles of problem and project based learning the aalborg model. Aalbord University (2010)Teichler, U., Schonburg, H.: editors. Comparative Perspectives on Higher Education and Graduate Employment and Work Experiences from Twelve Countries. Kluwer Pub. (2004)Ulrich, I., Borenstein, J.: Vfh+: reliable obstacle avoidance for fast mobile robots. In: Robotics and Automation, 1998. Proceedings, volume 2, pp 1572–1577 (1998)Verner, I., Waks, S., Kolberg, E.: Educational robotics An insight into systems engineering. Eur. J. Eng. Educ. 24(2), 201–212 (1999)C.y.A. Vicerrectorado de Estudios: Dimensiones competenciales upv (2013). http://www.upv.es/contenidos/ICEP/info/DimensionesCompetenciales.pdfWampler, C.W.: Manipulator inverse kinematic solutions based on vector formulations and damped least squares methods. IEEE Trans. Syst. Man, Cybern. 16(1), 93–101 (1986)Weinberg, J., Yu, X.: Robotics in education Low-cost platforms for teaching integrated systems. Robot. Autom. Mag. IEEE 10(2), 4–6 (2003

Smooth Three-Dimensional Route Planning for Fixed-Wing Unmanned Aerial Vehicles With Double Continuous Curvature

This paper presents a smooth flight path planner for maneuvering in a 3D Euclidean space, which is based on two new space curves. The first one is called 'Elementary Clothoid-based 3D Curve (ECb3D)', which is built by concatenating two symmetric Clothoid-based 3D Curves (Cb3D). The combination of these curves allows to reach an arbitrary orientation in 3D Euclidean space. This new curve allows to generate continuous curvature and torsion profiles that start and finish with a null value, which means that they can be concatenated with other curves, such as straight segments, without generating discontinuities on those variables. The second curve is called 'Double Continuous Curvature 3D Curve (DCC3D)' which is built as a concatenation of three straight line segments and two ECb3D curves, allowing to reach an arbitrary configuration in position and orientation in the 3D Euclidean space without discontinuities in curvature and torsion. This trajectory is applied for autonomous path planning and navigation of unmanned aerial vehicles (UAVs) such as fixed-wing aircrafts. Finally, the results are validated on the FlightGear 2018 flight simulator with the UAV kadett 2400 platform

New virtual laboratories for control applications

[Resumen] Este trabajo presenta dos nuevos laboratorios virtuales para reforzar la enseñanza e investigación en aplicaciones de control. El primero consiste en un sistema de control digital de un levitador magnético y el segundo un sistema de control digital de un aeropéndulo. Mediante los laboratorios virtuales propuestos es posible diseñar y validar distintas estrategias de control por computador mediante un interfaz de usuario sencillo e intuitivo basado en navegadores web. Aparte de la flexibilidad y autonomía en el uso de laboratorios virtuales, la propuesta contribuye a enriquecer la variedad de trabajos y prácticas de laboratorio. Además, permite mejorar la comprensión de los sistemas de control reales y evaluar el impacto de otros fenómenos en su comportamiento, como dinámica no modelada, retrasos de comunicaciones, etc.[Abstract] This work presents two new virtual laboratories to reinforce teaching and research in control applications. The first consists of a digital control system of a magnetic levitator and the second a digital control system of a hovercraft. Through the proposed virtual laboratories one can design and validate different control strategies by computer through a simple and intuitive user interface based on web browsers. Apart from the flexibility and autonomy in the use of virtual laboratories, the proposal contributes to enrich the variety of work and laboratory practices. In addition, it allows to improve the understanding of real control systems and to evaluate the impact of other phenomena on their behaviour, such as unmodeled dynamics, communication delays, etc.Ministerio de Ciencia e Innovación; PID2020-116585GB-I0